Skip to content

Report failed secret ranked submissions#508

Merged
msaroufim merged 1 commit into
mainfrom
badelsteinlelbach
Jul 5, 2026
Merged

Report failed secret ranked submissions#508
msaroufim merged 1 commit into
mainfrom
badelsteinlelbach

Conversation

@msaroufim

Copy link
Copy Markdown
Member

Summary

  • mark background ranked jobs as failed when public execution completes but required secret validation does not qualify the submission
  • include a clear job error explaining that the submission will not appear on the leaderboard
  • add regression coverage for a public-passed, secret-failed ranked submission

Root Cause

Background submissions treated successful runner completion as job success. Ranked submissions can complete runner execution while failing secret validation, leaving the leaderboard correctly unscored but the job status misleadingly marked succeeded.

Impact

Users polling submission status now see a failed job with an actionable secret-validation message instead of succeeded with no leaderboard score.

Validation

  • uv run --extra dev ruff check src/libkernelbot/background_submission_manager.py tests/test_background_submission_manager.py
  • uv run --extra dev pytest tests/test_background_submission_manager.py

DB-backed backend/leaderboard tests were attempted locally but Docker was unavailable (/Users/mark/.docker/run/docker.sock missing), so those should run in CI.

@msaroufim msaroufim marked this pull request as ready for review July 5, 2026 02:30
@msaroufim msaroufim merged commit 9c4026c into main Jul 5, 2026
4 checks passed
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/libkernelbot
  background_submission_manager.py 50, 55, 64-70, 85-86, 89, 99, 111, 394-395
  utils.py
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant